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TOOLKIT MODULE 1 
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TOOLKIT MODULES 


Module 1 defines and describes opportunistic experiments (OE) and highlights several case studies. 


Module 2 outlines key steps in recognizing potential OEs and assessing their feasibility. 


Module 3 provides an overview of the key steps in conducting an OE. 


I. Making the Case for Opportunistic 
Experiments 


Randomized controlled trials (RCTs) are considered the gold standard for evaluating 
an intervention’s effectiveness. RCTs provide the rigorous evidence of efficacy that 

is essential for policymakers and administrators to make evidence-based policy and 
program decisions. As human services agencies continue to face limits in resources 

and an increased focus on accountability, timely evidence-based decision making has 
become increasingly important in the policy area of family self-sufficiency and stability. 
Through collaboration with a research team, state and local agencies can seize oppor- 
tunities to introduce an RCT into the normal course of program operations without 
the high cost or disruption to normal operations sometimes associated with large-scale 
RCTs. This lower-cost, minimally disruptive approach—sometimes known as opportu- 
nistic experiments—uses random assignment not only as a fair way to allocate access to 
a program or service, but also to generate strong and timely evidence that can support 
policy and program decisions and improvements. 


An opportunistic experiment (OE) is a type of RCT that studies the effects of an 
initiative, program change, or policy action that an agency or program plans or intends 
to implement. In contrast, traditional, large-scale RCTs often examine an intervention 
or policy action that is developed and implemented specifically for a research study. 
OEs often use existing administrative data to measure outcomes for existing program 
participants. This can make them less costly and burdensome than other RCTs, which 
typically must collect new data and recruit participants. OEs initiated by state and local 
agencies can facilitate the recruitment of study sites, the cooperation of program staff, 
and the collection of low-cost administrative records data to measure outcomes. For 
these same reasons, findings from OEs can often be observed in a shorter timeframe. 
OEs typically focus on near- and intermediate-term outcomes typically observable 
within a 12 month period. By embedding research in planned changes and using 
existing administrative data, agencies can seize an opportunity to generate rigorous 
evidence on the effectiveness of program changes. As a result, OEs can help to inform 
ongoing program improvements in a timely manner. 


A related evaluation technique—rapid-cycle evaluation (RCE)—can also be used to 
assess the effectiveness of program operations and services. While traditional RCTs 
and OEs exclusively rely on experimental (random assignment) designs, RCE may 
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use experimental or quasi-experimental evaluation designs (Box I.1). When RCE 

uses an experimental design, it may also be considered an OE. RCE is “rapid” because, 
like OEs, it focuses on relatively short-term outcomes, and because it relies primarily 
on the use of existing data to measure outcomes. The “cycle” in RCE refers to the use 
of evaluation in a continuous improvement framework—a series of iterative experi- 
ments to identify the relative effectiveness of business or program processes and service 
delivery approaches—whereby an agency makes incremental modifications to improve 
program effectiveness. In this toolkit we focus on the use of OEs in the policy area of 
family self-sufficiency and stability. Where relevant, we draw parallels with and provide 


examples of RCE. 


Rigorous Research Methods: Key Terms 


Randomized controlled trial (RCT): An RCT is a rigorous research experiment 
considered to be the ‘gold standard’ for evaluating whether an intervention is effective 
in causing a given outcome. In an RCT, study participants are assigned, at random, to 
either a treatment group that receives an intervention, or a control group that does 
not. Random assignment ensures that the two groups are similar at the beginning of a 
study, which allows differences in the two groups’ outcomes over time to be attrib- 
uted to the intervention. 


Opportunistic experiment (OE): An OE is a style of RCT in which an experiment is 
embedded into a planned initiative, program change, or policy action. OEs rely upon 
existing administrative data to produce findings in a quick turnaround and low-cost 
manner. 


Rapid-cycle evaluation (RCE): RCE is an evaluation technique that uses either an 
RCT or a quasi-experimental (“comparison group’) design to assess whether incre- 
mental changes in program operations or services cause an improvement in out- 
comes. RCE is “rapid” because it focuses on relatively short-term outcomes and relies 
on existing administrative data. RCE often goes a step further than an OE by using 
evaluation “cycles;” that is, an iterative approach in which programmatic changes are 
tested, modified, and tested again, and thereby are less “opportunistic” in nature. 


This toolkit is designed to help researchers, policymakers, and agency administrators 
recognize and conduct OEs. In this first part (Module 1), we describe the use of OEs, 
explore their key features, and highlight several case studies. In the second part (Mod- 
ule 2), we discuss key steps and issues to consider when recognizing an opportunity 
for an experiment and determining the feasibility of conducting it. In the third part 
(Module 3), we outline and explain the critical steps in conducting an OE. This toolkit 
was produced as part of the Advancing Welfare and Family Self-Sufficiency Research 
Project (Project AWESOME), funded by the Administration of Children and Fami- 
lies’ (ACF) Office of Planning, Research and Evaluation (OPRE) within the U.S. 
Department of Health and Human Services.! 


1 This guide on opportunistic experiments in family self-sufficiency and stability research is adapted from a recently 
published report in the education area: “Recognizing and Conducting Opportunistic Experiments in Education: A 
Guide for Policymakers and Researchers" (Resch et al. 2014). 
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OEs are common in a variety of settings. Fundraisers sometimes test different styles 

of messaging in their communications with donors and supporters. As an example of 
both an OE and an RCE, a recent presidential campaign embedded experiments in 

its emails to potential donors and supporters by randomly assigning different email 
messages to different groups on its mailing list (Green 2012; Issenberg 2012). The 
campaign then tracked how much money each group donated, determined which 
messages were most effective, and then used them for future appeals. Similarly, busi- 
nesses test website designs to learn which ones works best in order to increase sales and 
improve user experiences (Mayer et al. 2009; Linden 2006). In education research, OEs 
frequently assess how educational strategies and practices influence student and school 
outcomes. For example, the New York City Department of Education recently embed- 
ded an OE in a planned literacy pilot intervention (Resch et al. 2014; Anand 2013; 
New York City Department of Education 2013). Forty middle schools were randomly 
assigned to receive the new literacy program or to serve as a control group. Existing 
student achievement data will be used to assess key outcomes of interest. If the new 
program improves reading achievement, the district will have rigorous evidence to 
support the expansion of the program to other schools; if it does not, the district can 
modify the program and then re-test it, or it can call off the expansion. 


Government human services agencies also conduct OEs to learn how best to use scarce 
resources to better serve citizens. The Virginia Department of Social Services (VDSS) 
was charged with increasing participation in the Earned Income Tax Credit (EITC) pro- 
gram as a result of a recent policy directive. However, the most cost-effective mechanism 
to achieve this goal was unknown. Should the state send out letters to potentially eligible 
taxpayers? Ox, would a phone call to notify likely eligible individuals of the program 
work better? Rather than assume which approach would be more effective in increasing 
program participation, VDSS took advantage of this opportunity to test and compare 
several outreach strategies and gather rigorous evidence to build its outreach strategy 
(Beecroft 2012). By randomly varying the outreach method across a set of households, 
Virginia learned that a strategy combining phone calls and mailings to eligible house- 
holds increased EITC participation and made cost-effective use of state resources. 


Oftentimes administrators are faced with implementing new policies or procedures 
without knowing if the changes will be an improvement over past practice. As another 
example of an OE in a human services agency, the New York City Human Resources 
Administration (HRA) used an experiment to test new administrative procedures to 
increase the establishment of child support orders for children receiving cash assis- 
tance (Dinan 2013). HRA determined that the new procedures, which minimized the 
required court appearances for noncustodial parents, were not successful at improving 
the rate of child support orders being established. The new procedure also took longer 
than the original procedures. Because of its study, HRA was able to clearly and quickly 
establish that the procedures were unsuccessful in meeting the initiative’s key goals. 
Efforts within HRA to improve the procedures’ outcomes are ongoing. In both the 
Virginia and NYC examples, the agency was able to undertake its experiment quickly 
and inexpensively by using existing administrative records and focusing on short-term 
outcomes. 
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Despite the relative ease of OEs, agencies often miss such opportunities in the fam- 
ily self-sufficiency and stability policy area. In such cases, an agency may continue to 
use resources—or even increase spending—on an intervention that is not shown to 

be effective. As an example, consider a state that invests in a new, more intensive job 
search assistance intervention for Temporary Assistance for Needy Families (TANF) 
clients. Without a test of the new intervention’s effectiveness, the agency will not know 
whether and to what extent it may affect clients’ employment status. Even if clients’ 
employment outcomes improve over time, without randomly assigning clients, for 
example, as a new intervention is gradually rolled out, the agency will miss an oppor- 
tunity to generate rigorous evidence of its effectiveness. While it might be tempting to 
attribute an improvement in employment outcomes to the new intervention, it is not 
possible to do so without a valid “counterfactual condition” that represents what would 
have occurred in the absence of the intervention. Without generating data to guide 
their decision, the agency may continue to fund the intervention without knowing 
whether that money is well spent. 


WHAT ARE THE DEFINING FEATURES OF OPPORTUNISTIC EXPERIMENTS? 


OEs typically have one or more of the following features, which may reduce costs 
or minimize the disruption to service delivery, and generate rigorous findings more 
quickly compared with traditional, large-scale RCTs: 


e An opportunity to randomly assign participants to an intervention. In OEs, 
participants are randomly assigned to an intervention in a way that minimizes the 
disruption associated with random assignment for the entity being studied, as well as 
its participants (such as a local TANF agency and its clients). When random assign- 
ment is seen as a fair way to allocate limited resources and is planned in a way that 
does not cause major disruption, it is easier to randomly assign local agencies, cohorts 
of clients, or individual clients to treatment and control conditions. 


e Experimentation on a program or policy change initiated by the entity being 
studied. OEs typically take advantage of an intervention or policy change that is 
initiated by the entity being studied. State and local agencies may implement new 
services or use new procedures in the belief that these changes will improve client 
(or other) outcomes, but lack rigorous causal evidence to support this belief. When 
administrators and researchers build research into a planned program or policy 
change, they introduce little disruption for participants and can evaluate whether the 
change does improve outcomes. Because the program or policy will be implemented 
with or without the evaluation, researchers working with agency staff may have an 
easier time recruiting sites. For example, if a state agency is planning to pilot a new 
program in a subset of its county agencies, researchers can work with state agency staff 
to encourage counties to participate in the study and undergo random assignment. 


e Strong partnerships between researchers and agency policymakers and admin- 
istrators. When research plans align with a state or local agency’s programmatic 
plans, everyone can benefit. The state or local agency can help focus the research 
questions toward actionable results, and close participation of state and local admin- 
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istrators—prompted by their being invested in learning the findings—may also 
facilitate easier access to administrative data and enhanced cooperation of staff. If 
the OE is conducted by an internal party, the evaluation might require fewer clear- 
ances than traditional, large-scale RCTs, speeding up the timeline and reducing the 
disruption introduced by research. In Box I.2, we discuss the importance of close 
partnerships between researchers and both agency administrators and policymakers 
in identifying opportunities for research and conducting OEs (and other studies) 
with minimal disruption or added cost. 


¢ Potentially easy access to data on key outcomes. The agency may already collect 
data on relevant outcomes through administrative records. Such readily available data 
may make a study less disruptive for participating agencies, staff, and clients than is 
the case in a study for which researchers must collect additional data to measure key 
outcomes (for instance, through a survey). An OE typically does, but need not have 
this feature. For example, an OE might capitalize on an opportunity to randomly 
assign participants to receive an intensive service intervention but require additional 
data beyond those collected conventionally (such as survey data on personal and fam- 
ily outcomes of participants). 


Close partnerships between researchers and agency 
administrators and policymakers 


Ongoing, mutually beneficial partnerships between researchers and agencies can be 
critical to conducting rigorous OEs with little added cost or disruption. Some agen- 
cies may have a researcher or team of researchers on staff, which provides a natural 
opportunity for a partnership. If not, agencies may find a research partner by recruit- 
ing one from a local university, contacting a research organization, or reaching out 
to researchers who are conducting a study or collecting data in their state. Even prior 
to the initiation of any study, agencies that establish an ongoing “thought-partner” 
relationship with a researcher or team of researchers can foster a culture that sup- 
ports evaluation and facilitates the early identification of research opportunities. Such 
thought-partner relationships can be: 


e Established through regular lunch meetings or forums; 


¢ Held with staff from several offices within the agency, rather than just one (given that 
formal interoffice contact may not be frequent enough to identify research opportu- 
nities sufficiently early); or 


e Focused on sharing regular status updates, in which agency administrators and 
policymakers discuss planned initiatives and seek feedback from a research perspec- 
tive, and in which researchers share results from their latest work and help identify 
potential opportunities to generate rigorous evidence on the effectiveness of an 
agency's initiatives 


When an opportunity for experimentation has been identified, continuing commu- 
nication between researchers and agency administrators and policymakers can help 
ensure that a research opportunity aligns with the agency's research agenda and is 
feasible to conduct. From there, close relationships between researchers and agency 
officials can help ensure that an experiment minimizes disruption for local offices, 
staff, and clients by facilitating recruitment, random assignment, and data collection, 
analysis, and reporting. 
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WHAT PROGRAM SITUATIONS PROVIDE OPPORTUNITIES FOR 
EXPERIMENTATION? 


Three common situations can facilitate random assignment and the use of OEs with 
minimal disruption to participants—a randomized pilot, an oversubscription lottery, 
and a staggered rollout of a program change. In each of the situations below, random 
assignment is a fair way to allocate limited slots for an intervention in which there are 
limited resources. In this section, we briefly describe each of these situations, and high- 
light several case studies from recent research that are largely characteristic of OEs. 


Randomized pilot. To test a new intervention, strategy, or program change before 
deciding whether to implement it more broadly, an agency can randomly assign sites, 
or individuals within a single site, to the pilot (treatment) group or to the business-as- 
usual (control) group. 


Oversubscription lottery. When interest in or demand for an intervention exceeds 
capacity (oversubscription), agencies may use a lottery to randomly assign interested parties 
to either a treatment group (offered a spot in the program) or a control group (not offered a 
spot). Random assignment becomes a fair way to allocate access to a program or service. 


Staggered rollout. When an agency lacks the resources to roll out a program change 
for all sites or individuals simultaneously, or simply wishes to understand the effect of 
the change before implementing it broadly, a staggered rollout design may be used to 
study an intervention or a change made to it. In this case, the treatment group may 
receive the program change during the initial phase of an experiment, while the control 
group essentially is placed on a wait list to receive it during a second phase. The second 
group can then act as a control to the former group during the first phase of the evalu- 
ation, but not after. If early analyses reveal that the program change is not effective, 
then the rollout in control sites can be cancelled or modified. 


As an example of a pilot strategy implemented and tested in a staggered manner, 
consider a hypothetical situation in which a state’s Supplemental Nutrition Assistance 
Program (SNAP) aims to increase program participation among youth and young 
adults formerly in the foster care system. To test a new outreach strategy for increasing 
participation, the state may pilot test the new strategy by staggering its implementa- 
tion. The state can randomly select a set of local offices to implement it first and serve 
as the treatment group. The local offices not selected for the strategy essentially become 
the control group. If the study’s findings show a short-term positive effect (for example, 
on program applications in the treatment group), then the state could implement the 
new strategy more broadly. If there are not positive effects, then the state may wish to 
modify the strategy and re-test it. 


IN CONCLUSION: MAKING THE CASE FOR OPPORTUNISTIC EXPERIMENTS 


‘The first part of this toolkit on the use of OEs in family self-sufficiency and stability 
research has shown that these relatively low-cost and quick turnaround experiments 
can be embedded within an agency’s ongoing operations in a manner that provides 

strong and timely evidence of effectiveness. Agencies can seize opportunities to con- 
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duct experiments in cases in which the intervention being studied will take place with 
or without experimentation, where clients or sites can be randomly assigned with 
minimal disruption to agencies and their staff, where strong partnerships exist between 
the research team and agency administrators and officials, and where there is fairly easy 
access to data on key outcomes. Researchers can play a critical role in helping agency 
administrators and policymakers identify and conduct OEs and make evidence-based 
decisions related to family self-sufficiency and stability. 


In the next two parts of this toolkit we discuss the issues to consider when recognizing an 
opportunity for an experiment and determining the feasibility of conducting it (Module 
2), and then describe the key steps in conducting such an experiment (Module 3). 


A supported employment pilot intervention in Ramsey 
County, MN 


To improve paths to employment for hard-to-employ TANF recipients and their 
families, the Ramsey County, Minnesota, Workforce Solutions Department piloted an 
integrated service model known as Families Achieving Success Today (FAST). FAST 

was devised as a co-located partnership of several agencies to provide mental health, 
vocational rehabilitation, health care, and employment services to TANF recipients 
with disabilities. It was built on the successes of the Individual Placement and Support 
(IPS) supported-employment model for individuals with severe mental illness (Bond et 
al. 2008). FAST follows the core principles of the IPS model: finding competitive jobs 

in the community that fit participants’ needs, strengths, and interests; fully integrating 
health services with employment services; and using rapid engagement into job search 
approach to help participants find jobs. Ramsey County partnered with a research team 
to conduct a small-scale random assignment evaluation of the FAST pilot to assess 
whether it improved client access to services, increased program and services uptake, 
and increased employment and economic stability outcomes (Farrell et al. 2013).+ 


From April to December 2011, 389 eligible TANF clients were randomly assigned to 

a treatment or control group. Treatment group members were offered a spot in the 
FAST program, while control group members continued to receive the usual case 
management and employment services accessible through TANF. The research team 
leveraged a variety of existing administrative data sources to measure impacts and 
other key research questions, including an automated case management system, case 
review notes, the state's benefit eligibility system, and unemployment insurance wage 
records. Qualitative interviews with key program staff and site observations were also 
conducted to examine fidelity of program implementation. The study found, among 
other things, that FAST treatment group members earned significantly more than 
control group members in the year after random assignment. 


The FAST pilot exhibits characteristics of an opportunistic experiment. Ramsey County 
tested a program policy change using a random assignment design while maintaining 
business-as-usual services for control group families. This allowed the county agency 
to measure and assess impacts of the intervention against status quo service delivery. 
Using administrative data and the help of a small research team, the FAST pilot was 
rapidly rolled out and evaluated in the span of 16 months. A limited-scope qualita- 
tive study provided useful formative feedback and, coupled with the impact results, 
provided enough feedback to enable Ramsey County to continue services and expand 
the FAST intervention. 


1 This experiment was part of the TANF/SSI Disability Transition Project that tested innovative pilot interventions 
targeting TANF recipients with disabilities. The project was funded by the Administration for Children and Fami- 
lies within the U.S. Department of Health and Human Services, under contract with MDRC and MEF Associates. 
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CASE STUDY 2 


A TANF and child support co-location pilot in Washington, DC 


In 2005 and 2006, the District of Columbia (DC) implemented a pilot service delivery 
change to improve collaboration between its Child Support Services Division (CSSD) 
and Income Maintenance Administration (or TANF) and, in so doing, increase the 
number of child support orders. This initiative was funded via a section 1115 demon- 
stration waiver grant from the Administration for Children & Families, which required 
a local evaluation of the intervention. To improve CSSD performance by increasing 
the number of TANF clients who participated in child support interviews (the first step 
in establishing a child support order), three CSSD intake workers were co-located at 
one TANF office. DC partnered with a research firm, the Lewin Group, to advise on 
and conduct an implementation study and impact evaluation of the pilot test (Farrell 
et al. 2007). More than 900 eligible clients were randomly assigned to a treatment 

or control group during a one-year period, where treatment clients received child 
support services on site at the TANF office. This involved taking part in both their TANF 
and child support intake interviews during a single visit to the TANF office. In contrast, 
control group members followed the standard procedure for child support intake, 
which involved an in-person interview at the separately-located CSSD headquarters. 


Key data for the evaluation were collected from two existing sources: a baseline 
information form completed by all TANF applicants and the automated child support 
enforcement data system. Supplemental interviews and observations were conducted 
by the research partner to inform an implementation study. Outcomes were examined 
twice—at 6 and 12 months-—following a client's eligibility determination and random 
assignment. The analysis showed that the treatment clients were significantly more 
likely to complete several key steps in the child support order process as a result of 
the co-location of the CSSD and TANF staff. The pilot test also demonstrated signifi- 
cant increases in the number of paternity and child support orders established for 
treatment clients compared with control clients. 


This study has several characteristics of an opportunistic experiment. It was embed- 
ded in a planned service delivery change and utilized existing data sources to test 
impacts. Notably, data analysis early in the study identified that many treatment clients 
were not completing the child support interview following their TANF intake interview. 
DC quickly reordered the process such that the child support interview took place 
first, while clients were waiting for their TANF intake interview. This change largely 
resolved the issue and serves as an example of how data analysis during the early part 
of an experiment can help facilitate ongoing program improvement. 
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A lottery-based evaluation of an intensive coaching 
intervention for recovering substance abusers in the child 
welfare system in Illinois 


Successive reports by the U.S. Government Accountability Office (1994, 1998) identi- 
fied parental substance abuse as a contributing problem in more than two-thirds of 
active foster care cases nationwide. Access to appropriate treatment for substance- 
abusing parents is low, and only about one-third of those who need substance-abuse 
treatment receive it (SAMHSA 1997). In response, several states have undertaken 
experimental demonstrations of integrated child welfare and substance abuse service 
models using Title IV-E waivers, which allow states to bypass financing regulations and 
test strategies for better serving families and children through integrated care (Ryan et 
al. 2006). The Illinois Department of Children and Family Services (IDCFS) and Office 
of Alcoholism and Substance Abuse implemented one such Alcohol and Other Drug 
Abuse (AODA) Waiver Demonstration Project in April 2000 to test a model of inten- 
sive case Management using substance abuse recovery coaches. The demonstration 
focused on two outcomes—improving access to substance abuse services and family 
reunification—over a three-year period from April 2000 to March 2003. 


IDCFS leveraged its existing program infrastructure to support the demonstration. 
Eligible families were screened for substance abuse at regular Juvenile Court Assess- 
ment Program (JCAP) custody hearings. Upon receiving a referral for substance 
abuse treatment, families were randomly assigned to a treatment or control group. 
The control group received services as usual, while the treatment group received all 
standard services plus intensive case management delivered via a recovery coach. A 
total of 738 families, representing 1,417 children, took part in the study. The evaluation 
used several administrative data sources from the involved agencies to measure out- 
comes. The demonstration resulted in a significantly higher uptake of substance abuse 
services and a significantly faster entry into services among treatment families relative 
to control families (Ryan et al. 2006). Moreover, a significantly higher fraction of treat- 
ment families (12 percent versus 7 percent of control families) achieved parent-child 
reunification. 


This study exhibits several characteristics of an opportunistic experiment. To increase 
interagency cooperation to improve program outcomes, IDCFS implemented a 
targeted service delivery change. The experiment leveraged the existing program 
infrastructure to carry out the intervention and administrative data to measure key 
outcomes. To address capacity issues, a lottery-style approach was used to randomly 
assign families to treatment or control groups. The study limited its scope by assessing 
two key outcomes for the intervention as a whole. This demonstration also laid the 
groundwork for a follow-on evaluation of the model's impact on substance-exposed 
births among families receiving recovery coaching (Ryan et al., 2008). 
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Il. Recognizing Potential Opportunistic 
Experiments and Assessing Their Feasibility 


In the face of limited resources and increased program accountability, human services 
agencies continue to invest in research that supports evidence-based policy and pro- 
gram decision making. Through collaboration with a research team, state and local 
policymakers and agency administrators can introduce an “opportunistic experiment 
(OE)” into normal agency operations by embedding a research experiment into a 
planned initiative, program change, or policy action. This relatively quick, low-cost 
evaluation approach can generate strong and timely evidence to inform policy and 
program decisions. 


In this section of the toolkit (Module 2), we describe three key steps and related issues 
to consider when determining both the potential for designing and the feasibility of 
conducting an OE. The first step is to look for opportunities when program or policy 
decisions may be made randomly, such as those in the case studies described in Module 
1 of this toolkit. Many situations in which a decision or choice is made are potentially 
opportunities to use random assignment. Once an opportunity for random assign- 
ment is identified, the next step is to consider the research and program questions of 
the state or agency and whether the potential evaluation aligns with these questions. 
‘The final step is to assess the feasibility of the evaluation. Integral to each step is a 


close relationship between researchers and agency administrators and policymakers (as 
discussed in Box I.2 in Module 1). 


STEP 1. LOOK FOR POTENTIAL RANDOM ASSIGNMENT OPPORTUNITIES 


In the usual course of program operations, certain situations can serve as opportuni- 
ties to allocate program services through a random assignment process. Programs, 
services, or processes can be allocated through a randomized pilot, an oversubscription 
lottery, or a staggered rollout of a program change (as described in Module 1). Agency 
administrators and policymakers should consider how the decisions and choices they 
make may create an opportunity to use random assignment to allocate services and 
thus provide an opportunity for an experimental study. Common situations in human 
services agencies may include: 


e Pilot test. An agency may be considering a new program or service delivery process 
but want to test the innovation before adopting it more broadly. State agencies may 
ask local agencies, for example, to volunteer to become pilot study sites. A random- 
ized pilot would provide the opportunity to rigorously test the innovation. The state 
agency could randomly assign the pilot and control sites from the population of local 
agencies that volunteer. 


e Limited resources. A state agency may be planning to introduce a new program 
enhancement or process change to all local agencies, but not have the financial or 
professional resources to implement the change in all local sites at once. This situa- 
tion lends itself to a randomized rollout. Randomly assigning participants to differ- 
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ent phases of the rollout may not be overly disruptive, and it will allow the agency to 
evaluate the change. 


e Excess demand. If a program or service is over-enrolled or is expected to have excess 
demand (perhaps due to limited resources), an oversubscription lottery can be used to 
assign slots. For example, in the education policy area, evaluations of charter schools 
and school vouchers have used a lottery mechanism to randomly assign students to 
overenrolled schools or oversubscribed voucher opportunities. In the area of family 
support and self-sufficiency, cases of excess demand may occur when new or a more 
intensive set of services are offered. For example, an agency may be offering new 
housing vouchers for low-income families or coaching and relationship education 
opportunities for child welfare and TANF clients. 


e Assignment of services. In the normal course of program operations, administra- 
tors may at times have the opportunity to assign individual clients or local sites to 
contracted service providers or specific types of services. In some cases, these assign- 
ments can be made randomly, thereby creating an opportunity for an experiment. For 
example, a TANF agency could randomly assign clients or local sites to one of two 
employment service providers. In this case, an OE could be conducted to compare 
the approaches and mix of services delivered by the two different providers. 


e Centralized communication. An agency may be trying to influence participant 
behavior through the centralized distribution of information—for example, by send- 
ing eligible families information on taking advantage of the EITC. If the agency 
wants to test the effectiveness of different strategies, an opportunistic experiment 
may be easier to implement if the communication effort is centralized. This can be 
viewed as a randomized pilot. 


STEP 2. CONSIDER HOW OPPORTUNITIES ALIGN WITH PROGRAM AND 
RESEARCH NEEDS 


While there are often potential opportunities for random assignment, before conduct- 
ing an opportunistic experiment, it is important to consider whether it can answer 

an important research question. When making this decision, consider whether the 
research question relates to an established research or program agenda or supports 

an underlying need or priority, such as increasing client participation in an activity 

or improving employment outcomes. Moreover, it is important to bear in mind that 
different policy questions will be answered by different study designs. For example, 
randomly assigning program participants to different treatment providers may provide 
information on which provider is more (or less) effective, but it will not provide an 
answer to questions about the overall effectiveness of the program. 


While addressing a previously identified priority may not be a prerequisite for the 
experiment, it is nevertheless important to consider the costs and potential disruption 
that may be incurred because of the experiment. Even relatively low-burden oppor- 
tunistic experiments entail costs for agencies and their research partners, and not all 
opportunities to use random assignment for an experiment may be an efficient use of 
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limited resources. The agency should thoughtfully consider how experimentation can 
best be used to inform and improve program practices and service delivery. 


STEP 3. ASSESS THE FEASIBILITY OF THE RESEARCH PROJECT 


As in all research, the next step is to consider whether the research study is feasible. 
‘These six questions can be used for determining the feasibility of a study: 


e What is the theory of change, and what comparison will be evaluated? 
e What are the specifics of random assignment? 

e What are the critical outcomes and data sources? 

e Is adequate statistical power possible? 

e Is research clearance needed? 


e What is the study timeline? 


In the remainder of this section, we examine each question in turn. Table II.1 high- 
lights the key issues and questions, described below, in assessing the feasibility of a 
potential research project. 


1. What is the theory of change, and what comparison will be evaluated? Before 
designing a research study, program administrators and researchers need to understand 
how an intervention works, consider who it will affect, and identify relevant outcomes. 
In other words, all evaluations of program interventions or process changes should 
begin with a theory of change. 


A theory of change describes the causal chain of events that leads from the implemen- 
tation of the intervention to the desired outcome. In some cases, the theory of change 
is simple—targeted GED tutoring and support is expected to directly influence the 
attainment of a high school credential—but in most cases it is more complex and thus 
helpful to create a diagram that captures the theory of change. This diagram includes 
the intervention, intermediate outcomes, and long-term outcomes, and it illustrates 
contextual factors that can affect the causal chain. Having a diagram provides structure 
for thinking about how the intervention affects long-term outcomes—or what actions 
or changes in behavior are expected to occur as a result of the intervention. If a theory 
of change does not exist, it is important to develop one, although the theory need not 
be a diagram. Researchers may need to lead development of the theory of change and 
then solicit feedback from the agency. 


In Figure 1, we present a simple theory of change for the TANF and Child Support 
Co-Location Pilot Project in Washington, DC, described in Case Study 2 in the first 
part of the toolkit (Module 1). The top row of boxes shows the typical path of TANF 
clients in the absence of the intervention as they receive services through the Child 
Support Services Division (CSSD). TANF clients are scheduled for a child support 
intake interview and can receive related child support services through the CSSD 
office. Following the arrows, some TANF clients will participate in a child support 
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interview and, over time, some will establish paternity or obtain a child support order. 
‘The black box in the second row represents the intervention, that is, the co-location of 
CSSD staff in a local TANF office. The remaining boxes in this row show the antici- 
pated results of the intervention— that more clients will actually participate in child 
support interviews and receive related child support guidance from the CSSD staff, 
thereby increasing the chances that TANF clients will complete additional steps in the 
process, establish paternity, and ultimately obtain a child support order. 


Theory of change “Business 
for the TANF as usual 
and Child Support aan ; a 
al H TANF GEIS Some clients Clune citans 
Co eno clients participate rexeT ad (ol) of-1 =) complete 
Project in Fequiredto in Cs in CS additional 
Washington, DC cooperate Yea intake nee 
with CSSD Tale-yau(-\war-le intortiotTs obtain CS 
CSSD office order 
cssp TANF clients More clients 
taff participate More clients complete 
GLE inCs participate FYelellateyar-lt 
(Xora (olet-]<-1e| F ‘ 
intake in Cs steps and 
at TANF Z : : é : 
office interview at interviews fo) eo) ¢-1 14m Ors) 
TANF office order 
Key contextual factors 
Client: socioeconomic status, interest in obtaining CS order 
CSSD staff: caseload size, child support order completion rate 
CSSD is Child Support Services Division 
CS is Child Support 
Figure 1 


A theory of change should explicitly differentiate between the proposed intervention 
(to which the treatment group is assigned) and current practice (to which the control 
group is assigned). In a research study comparing a new program with current practice, 
the study team refers to current practice as the “counterfactual condition”’—that is, 
what would have happened to members of the treatment group in the absence of the 
intervention. In Figure 1, the top row represents the counterfactual condition. The 
second row shows the anticipated causal path when the intervention is implemented. A 
comparison of the two rows illustrates the key research question. In this case, the study 
examines the impact of the intervention (shown in the black box) on the outcomes in 
the boxes on the right of each row. 


ADVANCING EVIDENCE-BASED DECISION MAKING: A TOOLKIT 


2. What are the specifics of random assignment? ‘The second feasibility consid- 
eration is to determine the specifics of random assignment. Random assignment can 
happen in different ways, so researchers should consider the following key questions: 


e Who or what is randomly assigned? The theory of change can help determine 


whether the appropriate level for random assignment is local sites, staff, and/or cli- 
ents. If the intervention is an agency-wide program, and if it would not be possible to 
find a control group of clients within a local office that is unaffected in some way by 
the intervention, then random assignment should apply to local offices, rather than 
individual clients.’ 


Consider how the level of random assignment is related to spillover, which refers 

to control group members possibly being affected by the intervention. For example, 
spillover may be a concern when rolling out agency-wide strategies if staff from 
different local offices interact regularly through professional events or intra-office 
work teams. In this way, the behavior of staff in the control group may be influenced 
by exposure to ideas or information from staff in the treatment group. Also consider 
how the level of random assignment is related to contamination, in which control 
group members actually receive all or part of the treatment. For example, in a lottery 
for an intensive intervention, the research team should closely monitor compliance 
with random assignment to ensure that clients assigned to the control group do not 
have access in any way to the treatment intervention. 


Who is eligible to be randomly assigned? Are they representative? In some 
opportunistic experiments, local sites or individuals may volunteer to be randomly 
assigned to treatment or control groups. They may be TANF clients wishing to enroll 
in a supported employment intervention, staff interested in a new training program, 
or local agencies willing to pilot a new program strategy. While randomly assigning 
volunteers does not affect the internal validity of a study, it can affect the generaliz- 
ability of study findings. In some studies, such as a study of a supported employment 
intervention, the impact on volunteers may be part of the key research question. 

In other words, policymakers may want to understand the impact of the intensive 
intervention on clients who are motivated to participate in the intervention. 


In other cases, such as when an agency is piloting a new Supplemental Nutrition 
Assistance Program (SNAP) application process, policymakers may be more inter- 
ested in the impact of the process on the average SNAP applicant, not its impact 
on applicants who choose to try the new process. The researcher must be aware of 
the implications of who is randomly assigned and interpret and present the results 
of the study accordingly. Likewise, it may help for researchers to provide agency 
administrators with examples of what a proposed study’s results would—and would 
not—enable them to say about a program's effectiveness. 


When and where in the service delivery process does random assignment 
occur? If random assignment occurs at one point in time and in one location, such 
as by a central administrator after all applications to an intervention are submitted, it 
can be fairly straightforward to implement. If the intake for an oversubscribed inter- 


2 The level of random assignment also has important implications for the data analysis. For details, see Murnane and 
Willett (2011). 
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vention occurs on a rolling basis in multiple locations, such as at each local agency, 
random assignment may be more difficult to implement and monitor because more 
staff will need to be trained on the procedures. 


e How big will the treatment and control groups be? Random assignment studies 
do not require that 50 percent be assigned to the treatment group and 50 percent 
to the control group. In lottery studies, the share of the participant pool assigned to 
the treatment group can vary widely on the basis of the amount of oversubscription. 
While the treatment—control ratio will have implications for statistical power (see 
the discussion of statistical power later in this section), other factors should also be 
considered. The number of treatment sites or individuals may depend on budget or 
capacity. For example, if an agency has limited funds to implement the intervention, 
it may choose to randomly assign only 30 percent of sites or individuals to the treat- 
ment group. 


Will random assignment be implemented with integrity? To ensure the integrity 
of experimental designs, compliance with random assignments must be monitored 
(and verified). In some cases, the researcher has control over compliance with random 
assignment, as when treatment sites receive tailored information that would not be 
useful to control sites, thus reducing the likelihood of spillover. In other cases, the 
researcher must monitor compliance with random assignment, as when monitoring a 
lottery process to ensure that lottery results are recorded accurately and that offers of 
intervention services are made only to those assigned to the treatment group. 


3. What are the critical outcomes and data sources? The third way to assess the 
feasibility of a research project is to consider ways to reliably measure the key outcomes 
of interest and assess the difficulty of obtaining the required data. A key prerequisite to 
an efficient opportunistic experiment is the availability of relevant and usable adminis- 
trative data with outcomes of interest that are observable in the data (Cody and Asher 
2014). Using outcome measures from administrative records data may be the most 
cost-effective way to collect outcome measures for opportunistic experiments. In Box 
II.1, we highlight data often available through state agencies related to an individual 
family’s self-sufficiency and stability. Although opportunistic experiments are often 
thought of as cheap and quick and thus focused on short-term outcomes, the need for 
long-term outcomes should not preclude a researcher from starting an opportunistic 
study. In Box II.2, we further explore long-term outcomes. 


When assessing the potential outcomes and data sources, consider the following questions: 


e Are the outcomes cost-effective to collect? It is important to be realistic about 
the resources necessary to evaluate impacts on outcomes of interest. With limited 
resources, evaluations may focus on a specific subset of outcomes that can be col- 
lected most easily. 


e Are the outcomes measured similarly for the treatment and control groups? 
If not, the evaluation will not be able to determine whether differences in outcomes 
are due to differences in effectiveness or differences in how outcomes are measured. 
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e Are the data complete and accurate? What are the patterns of missing data; that 
is, are certain individuals more or less likely to have missing data? If outcome data 
are frequently missing or missing disproportionately for one group, the outcome may 
not be appropriate for evaluation or, at the least, results should be used with caution. 
In some cases, agencies may wish to supplement administrative data on employ- 
ment, earnings, and other variables with data collected from a short survey, on-line or 
otherwise. A survey can be used, for example, to collect more detailed information on 
the characteristics of clients’ jobs or to capture whether clients are working in infor- 
mal jobs, out-of-state jobs, or self-employed jobs, which are not likely to be included 
in a state’s administrative data. 


Key outcomes from state administrative 
data and other sources 


Administrative data are an ideal source for outcome data in an opportunistic experi- 
ment. Human services programs like TANF, SNAP, and child support and child welfare 
programs track regular administrative data on client characteristics, program par- 
ticipation, and outcome data. In addition, unemployment insurance (UI) data from 
state departments of labor offer employment and earnings records data. In addition, 
many program administrators use readily available types of program-level data such 
as TANF work participation rates, SNAP error rates, and metrics for staff performance 
monitoring. It makes sense that when an opportunistic experiment tests for the effect 
of a program change (such as a more self-directed SNAP application process) ona 
measureable outcome (such as SNAP participation) to leverage existing administrative 
and program data capacities. 


While the specific data elements vary by state, states’ administrative and program data 
sets commonly include the key types of measures, below. Data sharing agreements 
between state agencies can facilitate a given agency's access to administrative data. 


¢ Client identifiers (important for linking data sets) 

¢ Client demographics and eligibility data 

e Program enrollment, attendance, and participation 

¢ Benefits received (in dollar values) for a given month or quarter 

e¢ Employment status in a given month or quarter 

¢ Earnings (in dollar values) in a given month or quarter 

¢ Child support processes and orders 

¢ Child welfare services, processes, placements, and parent-child reunifications 


¢ Child care, transportation, and other supportive service benefits received and 
amount in dollars 


¢ Staff data 
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Exploring long-term outcomes 


Agencies may be less eager to conduct an opportunistic experiment if the key 
outcome of interest is not available for several years. However, the need for long- 
term follow-up should not necessarily preclude agencies and their research partners 
from starting such a study. Intermediate outcomes may provide early evidence of an 
intervention’s success, and implementing random assignment preserves the validity of 
impacts on intermediate outcomes and options for future follow-up using administra- 
tive data or surveys. 


In the case of the TANF and Child Support Co-location Project (Case Study 2 and Fig- 
ure 1), for example, the ultimate goal of the co-location intervention was the longer- 
term outcome of increased child support orders for TANF clients. However, the study 
focused first on the key short-term outcome of client participation in the initial child 
support intake interview. The intervention’s theory of change assumed that some 
clients did not participate in the intake interview because the location of the interview 
was not convenient. Documenting the short-term impact on client participation in the 
intake interview was the first indication that the intervention might have had an impact 
on the ultimate long-term outcome of an increase in child support orders. Short-term 
outcomes were measured six months after random assignment, while longer-term 
outcomes, such as the establishment of paternity and a child support order, were 
measured 12 months after random assignment. 


4. ls adequate statistical power possible? The fourth feasibility consideration is 
whether the experiment will be able to detect an effect of a given size. Conducting a 
power analysis can help the research team determine how large a sample the opportu- 


nistic experiment will need to detect a policy-relevant effect with reasonable precision. 


Although researchers may not be able to control the sample size in an opportunistic 
experiment, they should consider statistical power analysis as a tool for determining 
whether the potential experiment can generate findings with adequate precision.? In 
some circumstances, such as in a state with a small number of local offices, it may not 
be feasible to evaluate a program implemented at the local office level. 


Several factors can affect an experiment’s statistical power:* 


e Size of the expected impact. If an intervention is expected to have a large impac 


t 


on outcomes, the impact will be easier to detect, which means that a smaller sample 


is required to detect that effect. Researchers should assess the size of the expected 
impact by considering the theory of change and examining prior research. Gener- 
ally, an intensive intervention is more likely to have a large impact than a “light 
touch” intervention. 


e Sample size. Statistical power generally improves with a larger sample size. 
Researchers should thus ensure that the estimated size of the research sample is 
sufficient to detect the expected impact. Box II.3 discusses how to proceed in cases 
when the sample size may be difficult to estimate. In studies using a rolling random 


3 For more details on statistical power analysis, see Murnane and Willett (2011, chapter 6). 
* For more details on calculating statistical power for random assignment studies, see Schochet (2008). 


21 


ADVANCING EVIDENCE-BASED DECISION MAKING: A TOOLKIT 


assignment process, researchers will need to consider how long it will take to accrue 
the necessary sample. 


e Level of random assignment. As mentioned above, statistical power improves 
with a larger sample size. However, if random assignment is conducted at the local 
agency level, the primary determinant of statistical power is the number of local 
agencies (sites), not the number of clients served by the local sites. In this case 
researchers should pay special attention to the number of local sites that can be 
included in the study. 


e Availability of baseline data. Baseline data on characteristics related to the outcome 
can improve statistical power by accounting for variation in outcomes that can be 
attributed to differences in these pre-random assignment characteristics. Prior mea- 
sures of the outcome—for example, food stamp receipt during the previous year—are 
especially valuable because they tend to explain a large share of the variation in the 
outcome measure (Bloom et al. 2007; Schochet 2008). 


Box II.3. Estimating the size of a research sample 


If the random assignment mechanism is a lottery, it may be difficult to estimate the 
size of the research sample. The primary factor in determining sample size is the 
number of applicants or potential clients in the lottery. Examining historical applicant 
trends may provide some guidance, but researchers should also consider other fac- 
tors that may affect demand for a program, including enrollment in similar or related 
programs, as well as social, demographic, and economic indicators in the community 
that may be suggestive of the potential demand for a program and/or the supply of 
eligible applicants. Programs may make modifications to a lottery process, such as 
exemptions for certain clients (for example, siblings or relatives of existing clients). 
These types of modifications should be considered when estimating the size of the 
research sample (Tuttle et al. 2012). See Box III.2 for more information on exemptions 
(exclusions) from random assignment. 


5. Is research clearance needed? ‘The fifth feasibility consideration is whether any 
research clearance is required. Research studies may be subject to multiple layers of 
clearance or review, depending on the details of the study and the policies of the par- 
ticipating states and agencies. Many states or agencies require researchers to complete 
a research application before undertaking a study. States and agencies may also require 
that researchers provide evidence of approval by an Institutional Review Board (IRB) 
or complete a data use agreement. The requirements can vary widely across states and 
agencies, but a close ongoing partnership between agency administrators, policymak- 
ers, and researchers can ensure an understanding of what is required and may lead 

to expedited review. Situations in which multiple layers of clearance and review are 
required may not be good candidates for opportunistic RCTs, because these reviews 
can be costly and time consuming. However, in many cases, an IRB clearance can be 
expedited for opportunistic experiments. 
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Research that involves human subjects may require approval from an IRB. An IRB 
review typically requires the researchers to present the research questions and plans 
for protecting the participants. There are state and local variations in requirements, but 
all research studies with human subjects are not required to go through the full IRB 
process. Studies viewed as posing minimal risk to human subjects may be exempted or 
eligible for expedited review. 


A data-use agreement defines the terms under which researchers may access certain 
data that have been collected by another entity. Data-use agreements are usually 
required before external researchers can gain access to administrative records data. 
Such agreements help to ensure data confidentiality and data security, and prohibit the 
unlawful disclosure of personally identifiable information. The handling of data agree- 
ments varies by state. For example, some states provide researchers with confidential- 
ity and data security restrictions, some states provide only identified data, and others 
require researchers to work onsite (or any combination of these scenarios). 


Data-use agreements can also describe plans for the dissemination of study results and 
may specify the types of publications that would result from the study, such as public 
reports, issue briefs, or articles submitted to a peer-reviewed journal. Data-use agree- 
ments can specify, if requested, that the agency providing the data be kept anonymous. 
Agreements often grant the data-providing agency the right to review all publications 
for compliance with the agreement’s term. 


6. What is the study timeline? The final feasibility consideration is a matter of 
timing. As in many other research endeavors, the feasibility of an opportunistic experi- 


ment depends critically on the study timeline and whether researchers can implement 
random assignment at a time that works with the agency’s decision-making calendar 
and other program priorities. Participation by the agency in another research study at 
the same time may also complicate its involvement and introduce additional burdens 
that make the new study prohibitive. 


A second timing consideration has to do with results. The nature of the research ques- 
tions asked in opportunistic experiments means that practitioners and policymakers 
want to know the results of the study quickly so that they can make decisions based 
on those results. Researchers should consider these interests while also being realistic 
about the study timeline. Factors that may affect the timeline include the need to 
secure outside funding, time to recruit study participants, and time to negotiate access 
to data. When results are needed quickly, the opportunistic experiment may still be a 
feasible option if researchers focus on only short-term outcomes. 
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1. What is the theory of change? 


¢ Can it explain the causal chain of events that leads 
from the implementation of the intervention to the 
desired outcome? 


e Is it explicit about how the proposed intervention is 
different from current practice? 


2. What are the specifics of ran- 
dom assignment? 


e Who or what is randomly assigned? 
e When and where does random assignment occur? 
¢ How big will the treatment and control groups be? 


e Will random assignment be implemented with 
integrity? 


3. What are the critical outcomes 
and data sources? 


e Are the outcomes cost-effective to collect? 


¢ Are the outcomes measured similarly for the treat- 
ment and control groups? 


e Are the data complete and accurate? 


4. Does the experiment have suf- 
ficient statistical power to answer 
the questions of interest? 


e What is the size of the expected impact? 
e What is the sample size? 

e What is the level of random assignment? 
e Are baseline data available? 


5. Is research clearance needed? 


¢ Does conducting random assignment or accessing 
outcome data require approval from an Institu- 
tional Review Board or a data-use agreement? 


¢ How can a research partner help facilitate the 
completion of these steps? 


6. What is the study timeline? 


e Is the opportunity for random assignment time- 
sensitive? 

¢ Does the timeline depend on securing outside 
funding, recruiting study participants, or negotiat- 
ing access to data? 


e Are the results of the study needed for immediate 
decision making? 
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Ill. Conducting Opportunistic 
Experiments: Key Steps 


IDENTIFY ENCOURAGE 
SITES PARTICIPATION 


CONDUCT & 
MONITOR 


(ove) HA 04 
DATA 


ANALYZE & 
REPORT 


By embedding a randomized controlled trial (RCT) into a planned initiative, program 
change, or policy action, agency administrators, policymakers, and researchers can work 
together to conduct a quick turnaround, low-cost evaluation. Such an “opportunistic 
experiment (OE)” can generate strong and timely evidence to inform policy and pro- 
gram decisions. In this third section of the toolkit (Module 3), we provide an overview 
of five key steps that researchers, alongside the administrators and policymakers 

with whom they collaborate, can follow to conduct an OE. All experiments, whether 
considered opportunistic or not, include these steps: (1) identify sites; (2) encourage 
participation in the study; (3) conduct random assignment and monitor compliance; 
(4) collect data; and (5) analyze data and report findings. We focus on aspects of these 
steps that may be somewhat different in OEs than in traditional RCTs. We draw on 
examples, both real and hypothetical, to illustrate the key steps and related issues. Table 
III.1 highlights the various steps and issues, described below, in conducting an OE. 


STEP 1. IDENTIFY SITES 


In many opportunistic studies, the plans for the intervention determine the set of pos- 
sible sites. This can be an advantage, because the researcher can focus immediately on 
the relevant sample. For example, in a hypothetical pilot of a new SNAP application 
process in a given state, the set of potential sites may be county offices in the state that 
are not already using the new application process. An opportunistic experiment with 

a well-defined set of possible sites may be more cost-effective because the researchers 
will require fewer resources for recruiting sites. 


STEP 2. ENCOURAGE PARTICIPATION 


In some cases, experiments do not require recruitment of sites or participants, as in 
instances of centralized communication (where participants do not need to agree 

to participate in the intervention) or oversubscription (where participants may have 
already agreed to participate in the intervention but there aren't enough slots). Where 
recruitment is necessary, opportunistic experiments still offer advantages. First, if 
sites are already planning to implement an intervention or are eager to implement it, 
researchers do not have to sell the study to sites (explaining how the benefits of the 
intervention outweigh the disruption that comes with it). Second, if the research relies 
on administrative data, the experiment does not require data collection that disrupts 
program practices, which may make it easier to recruit participants. Where random 
assignment is added for research purposes, the research team will need to work with 
the state, agency, or local sites that implement the intervention to encourage partici- 
pants to undergo random assignment (see Box III.1). 
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Depending on what or who is being randomly assigned, the research team may work 
with administrators from the state or local agency to ensure that assignment to treat- 


ment and control groups is truly random. Three key issues to consider when conducting 


and monitoring compliance with random assignment include: 


Encouraging participation in research that 
uses random assignment 


Program administrators may hesitate to randomly assign local sites or prospective 
participants to treatment conditions for many reasons, including ethical or logisti- 
cal concerns (Cook 2001). The following strategies can address these concerns and 
encourage administrators and agency staff to participate: 


e Emphasize that because the agency is planning to implement the intervention with 
or without a study, it is in their best interest to know whether the intervention works. 
An experiment can enable an agency to discontinue use of an ineffective interven- 
tion or expand use of an effective intervention. 


¢ Name the study's potential benefits, such as enabling agencies to better serve clients, 
better allocate limited resources, and better use taxpayer dollars by continuing the 
use of effective interventions only. 


¢ Discuss the ramifications of making decisions without rigorous evidence. 


¢ Convey that in the case of oversubscription or limited resources (or both), random- 
ization is a fair and transparent way to distribute resources and services. 


¢ Assure participants that the study has been designed to minimize disruption for local 
sites, staff, and clients. 


¢ When necessary, allow for a limited number of exemptions from random assignment. 


¢ If needed, the agency can offer incentives, such as small gift certificates, to encour- 
age participation. 


¢ Ensuring the system for assignment is truly random. Where agencies have sys- 
tems for assigning sites or individuals to a program or intervention, the research 
team should ensure that the systems are truly random. For example, some systems 
use methods that administrators describe as “effectively random” (such as assigning 
clients to a program by the alphabetical order of their names), which the research 
community does not typically recognize as random because the methods may result 
in systematic differences among groups.’ In such cases, the research team should 
work with agency officials to agree on a truly random mechanism for assignment. 


e Accommodating agency needs and preferences for random assignment. ‘The 
research team may also need to work with agency officials to accommodate needs 


and preferences that influence random assignment. For example, in an opportunistic 


experiment in which clients are randomly assigned to case managers, researchers 


5 Effectively random methods may not meet evidence-based guidelines for random assignment and methodological 


quality if it is unclear that all units in the study were assigned by chance (U.S. Department of Education 2005). 
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can work with agency staff to accommodate preferences for caseload composition 
(such as a balanced level of client need across case managers).° Likewise, if an agency 
needs to roll out a program to only a few sites at a time due to resource constraints, 
researchers can conduct a randomized staggered rollout. Researchers and staff must 
identify any exceptions (exclusions) in advance of random assignment. In Box III.2, 
we discuss how researchers can accommodate exclusions from random assignment. 


Accommodating exclusions from random assignment 


Just as in other randomized controlled trials, researchers may face pressures to 
include certain sites or individuals in the treatment group of an opportunistic experi- 
ment. One strategy to alleviate this pressure is to exclude some sites or individuals 
from randomization (and the study), but let them participate in the intervention. For 
example, staff may strongly prefer that a particular individual receive the interven- 
tion (such as a client nearing the end of a TANF time limit), so random assignment to 
the control group would be unacceptable. In such instances, these individuals could 
receive the treatment but not be included in the study. 


To preserve both sample size and power, as well as the ability to generalize results 

to other settings, researchers should minimize such exclusions. When exclusions are 
necessary, researchers and agency staff should discuss and negotiate eligibility for 
exclusion from random assignment (and thus from the study) well before random 
assignment occurs. As in any study, research teams should work with agency adminis- 
trators to keep accurate documentation of any exclusions. 


e Monitoring and confirming compliance with random assignment. After conduct- 
ing random assignment, researchers should work with staff to monitor and confirm 
compliance with random assignment so that no changes are made after assignments. 
The monitoring task for opportunistic experiments is similar to that for other RCTs: 
when relevant administrative records are available, researchers should verify that the 
actual assignments of local sites or individual clients to conditions match the origi- 
nal random assignment. Even in an oversubscription lottery for which the agency 
is using random assignment to regulate admission to oversubscribed programs, the 
researcher must still monitor random assignment and compliance. In Box III.3, we 
outline how researchers can ensure validity of randomization generated by lottery. 


Box III.3. Monitoring randomization in a lottery study 


In lotteries for oversubscribed programs, attention to recordkeeping and compliance 
with the randomized list may vary across local sites. To ensure validity of randomiza- 
tion generated by lottery, the researcher should ideally: 


¢ Participate in the lottery by observing or even generating the randomized lists. 
« Compare the randomized lists to actual program offers and enrollment records. 


® For example, researchers may pairwise match the research sample on relevant site or client characteristics when planning and 
conducting random assignment to ensure balance on key characteristics. After researchers have constructed the pairwise 
matches, sites or clients are randomly assigned within pairs (often using a random number generator) to treatment conditions. 
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In some opportunistic experiments, all relevant data may be readily available (such as 
through various administrative sources), but others will require additional data col- 
lection. To preserve the possibility of longer term follow-up, researchers could collect 
individual identifiers, such as name, birthdate, address, and Social Security number. 
With this information, researchers may find sample members in other administra- 
tive data sources or contact them for a follow-up survey if funding for such research 
becomes available. However, these data can be difficult for researchers to collect 
because of privacy protections, and the collection of these data typically must be 
included in a data-use agreement. 


STEP 5. ANALYZE DATA AND REPORT FINDINGS 


Analysis and reporting activities are similar across opportunistic experiments and other 
RCTs. Researchers should conduct the same careful analyses that they would in any 
study, accounting for such issues as low treatment take-up, control group reactions 
caused by anticipation of treatment, spillover of the intervention to the control group, a 
lack of baseline data, and attrition.’ 


Disseminating the findings from opportunistic experiments, as well as the context in 
which the experiment took place, is extremely important. When disseminating find- 
ings, researchers should describe what is being compared with the intervention. It is 
much more informative, for example, to report that a job search assistance interven- 
tion improved employment-related outcomes relative to the existing program than to 
just describe the pilot as effective. Researchers can also emphasize that dissemination 
not only contributes important information to the field but also helps the agency and 
other stakeholders build an evidence base relevant to their context and disseminate the 
results to administrators and staff who are not involved in the study. 


IN CONCLUSION 


At every level of the human services system, leaders need to know which programs, 
policies, and practices are most effective as they allocate scarce resources. The three 
modules that comprise this report offer a toolkit to guide researchers, agency admin- 
istrators, and policymakers as they work together to identify, plan, and conduct RCTs 
related to family self-sufficiency and stability. Relatively low-cost and quick turnaround 
opportunistic experiments can be embedded within an agency’s ongoing operations in 
a manner designed to provide strong and timely evidence of effectiveness. Such studies 
allow agencies to learn about an intervention’s effectiveness and make evidence-based 
decisions to support families as they work toward self-sufficiency and stability. 


®Murnane and Willet (2001) and the What Works Clearinghouse guidelines (U.S. Department of Education 2011) provide 
guidance on how to account for these issues in analysis. 
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Key steps and Step 1: Identify sites ¢ What is the set of possible study sites? 

questions 7” Step 2: Encourage participation e Are agencies already planning to implement the 

conducting an intervention or are eager to implement it? 

opportunistic e Are there any incentives or benefits to the agency 

experiment that can be used in recruiting sites and/or partici- 
pants? 


¢ If needed, will state and local staff implementing 
and overseeing the intervention be able to work 
with the research team to encourage participants 
(either site-level or individual participants) to 
undergo random assignment? 


Step 3: Conduct and monitor ¢ Does the research team need to work with agency 
random assignment administrators or program staff to ensure that the 
system for assignment is truly random? 


¢ Does the research team need to work with agency 
administrators or program staff to accommodate 
nonnegotiable needs and preferences in the ran- 
dom assignment process (such as balanced levels 
of need across sites)? 

Will agency administrators or program staff be 
able to help researchers accurately document any 
exclusions? 

Will agency or program administrators be able to 
help researchers monitor and confirm compliance 
with random assignment to ensure that no changes 
are made after generating assignments? 


Step 4: Collect data Are all relevant data readily available (such as 
through various administrative sources), or is there 


a need to collect additional primary data? 


When possible, will administrators or staff be able 
to help researchers obtain baseline and outcome 
data? 

To preserve the possibility of longer term follow- 
up, will administrators or staff be able to help 
researchers collect individual identifiers, such 

as name, birthdate, address, and Social Security 
number? 


Step 5: Analyze data and report Will administrators or staff be able to help 

findings researchers conduct the same careful analyses that 
they would in any study—accounting for issues 
such as low treatment take-up, control group reac- 
tions caused by anticipation of treatment, spillover, 
a lack of baseline data, and attrition? 


To generate knowledge for the family self-suffi- 
ciency and stability community, will administrators 
or staff be able to help researchers report and dis- 
seminate findings from opportunistic experiments? 


Table III.1 
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